VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability evaluated the object captioning challenge (nocaps). In this challenge, no additional image-caption other than COCO Captions, allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pre-training (VIVO) performs absence of caption annotations. By breaking dependency paired data VLP, VIVO leverage large amounts image-tag learn visual vocabulary. done by multi-layer Transformer learns align image-level tags with their corresponding region features. To address unordered nature tags, uses Hungarian matching loss masked tag prediction conduct pre-training. We validate effectiveness fine-tuning pre-trained captioning. addition, we perform an analysis visual-text alignment inferred our model. The results show not only fluent objects, but also identify locations these objects. Our single has achieved new state-of-the-art on nocaps and surpassed human CIDEr score.